feat: embed ethrex as the execution layer (in-process) - #530

A 3-node devnet with the embedded EL never finalized: 22 `parent_hash mismatch`
errors, peer imports halved, no aggregation coverage. A consensus-only control run
on the same image finalized normally, isolating the fault to the integration.

The state transition requires

    payload.parent_hash == state.latest_execution_payload_header.block_hash

which is the parent the consensus chain expects. build_payload instead derived it
from store.get_latest_canonical_block_hash(), the node's *own* EL head. Each node
runs an independent in-memory execution layer, so those two drift apart, and a
proposer's payload then named a parent no peer agreed with — every node's STF
rejected the block. It failed silently from the proposer's side: no EL rejection,
no warning, the block just never stuck.

The Engine-API path did not have this bug, because its build-mode
forkchoiceUpdated pointed the EL at el_hash_at(store.head()) before building.
Collapsing that two-step into a single call dropped the step that chose the parent.

build_payload now takes parent_el_hash explicitly and re-points the EL at that
block before building; el_integration passes el_hash_at(head_root). safe and
finalized are left unset in that fork-choice call, since pinning them would forbid
a later build on an earlier block.

Regression test builds_on_the_requested_parent_not_the_el_head advances the EL two
blocks, then builds on block 1 and asserts the payload names that parent; it fails
against the previous code. Unit tests could not have caught this — it needs more
than one execution layer to appear.

Verified: 3 nodes, 36 slots, finalized at slot 40 with all nodes on the same
finalized root and 43 payloads executed each; zero parent_hash mismatches, zero
synthetic fallbacks, zero panics. Matches the consensus-only control (slot 41).

… layer

Until now nothing could put a transaction into the system. ethrex is linked as a
library and the node deliberately does not depend on ethrex-rpc, so there is no
eth_sendRawTransaction, and add_transaction_to_pool had no caller. Every block
the chain has ever produced was empty.

POST /lean/v0/admin/el/tx takes {"raw": "0x02f8..."} and returns
{"tx_hash": "0x..."}. It is routed under /admin because it is a node-operator
affordance rather than part of the Lean consensus API. ethrex validates the
transaction itself — encoded size, duplicate hash, signature recovery, nonce,
balance, chain id, replacement rules — so the handler adds no validation of its
own and passes the mempool's rejection message through verbatim, which is the
part that tells a caller why their transaction did not land.

A node started without --el-genesis answers 501 rather than omitting the route,
so the difference is visible to a client instead of looking like a routing
mistake. That is deliberately not the 503 the aggregator endpoints return for a
missing controller: 503 invites a retry, whereas an execution layer is configured
at startup and will never appear.

The engine reaches the handler as an axum Extension, matching how the aggregator,
sync-status and event-bus handles are already threaded through start_rpc_server.
No trait and no new abstraction: ethlambda-ethrex-engine was already in this
crate's dependency graph via ethlambda-blockchain, so depending on it directly
only makes an existing edge explicit.

The devnet script now exercises the whole path. Four slots after genesis it posts
a signed transfer to every node, then scans the chain for those exact raw bytes
and asserts the containing block reports gasUsed > 0. It submits to every node on
purpose: without execution-layer gossip a transaction sits only in the mempool
that received it, so submitting to one node means waiting for that node's turn to
propose. Fanning out means whichever node proposes next includes it, and the
others evict it when they import the block.

The transaction is a checked-in hex fixture rather than signed at runtime, since
bash cannot sign and the RPC crate should not need secp256k1 just to build a test
input. ethlambda-ethrex-engine gains an ignored regenerate_rpc_fixtures test that
writes it, so it is reproducible rather than an opaque blob; rerun it if the EL
genesis chain id or funded account changes.

Verified: 575 tests pass, 10 of them new, covering the happy path, hex with and
without the 0x prefix, idempotent resubmission, the no-execution-layer case, and
five malformed-input cases. fmt and clippy -D warnings clean. On a 3-node devnet
all three nodes accepted the transaction, it was included in the first block
built after submission, and that block reports gasUsed=21000 with a receipts root
that differs from the empty-trie root.

A transaction submitted to one node reached only that node's mempool, so it sat
there until that particular node's turn to propose. On a 16-node devnet that is up
to 16 slots of waiting, and the workaround — submit to every node — does not scale
past a devnet the submitter happens to control.

Consensus already replicates *execution*: every block carries its payload, so
peers re-execute it locally. What consensus cannot carry is a transaction that has
not been included yet. This gives the execution layers their own mesh for exactly
that, using ethrex's devp2p (discv4 discovery + RLPx over TCP) alongside, and
entirely independent of, the consensus layer's libp2p. Separate key, separate
port, separate peer set. No changes to ethrex were needed; its public API is
sufficient at the pinned revision.

--el-p2p-port enables it and --el-bootnodes seeds discovery. One enode is enough:
discv4 finds the rest of the mesh from there, and each node logs its own at
startup.

Four settings differ from ethrex's defaults, each for a reason:

set_synced() is called after the stack is up. ethrex gates inbound Transactions,
NewPooledTransactionHashes and PooledTransactions on is_synced(), which defaults
false and is otherwise only set by the Engine-API fork-choice handler this
integration bypasses. Without it every inbound transaction is dropped *silently*,
which looks exactly like a mesh that never formed. It is correct for a
consensus-driven execution layer, which is never behind in the sense the flag
means, and no syncer is reachable from a P2PContext so it cannot start a snap
sync. Set in start_p2p rather than at construction, so a node running without
execution-layer gossip does not claim to be synced.

target_peers is 8 rather than 100. On a devnet every node is a candidate, so a
target above N-1 means the peer table is never full and every node keeps dialling
every other one forever. Nothing in ethrex sends DisconnectReason::AlreadyConnected
and new_connected_peer overwrites its entry, so simultaneous dials in both
directions leave duplicate connections carrying duplicate transaction traffic.

discv4 is on and discv5 is off. Discovery cannot be disabled outright: with both
off, start_network discards the bootnode list entirely, which would force every
node to know every other node's enode up front. discv5 would add a second protocol
surface and find nothing discv4 does not.

The node binds 0.0.0.0 but advertises 127.0.0.1. Advertising the bind address
would propagate 0.0.0.0 over discv4 and every dial back would fail.

The execution layer's key is derived from the consensus node key as
keccak256("ethlambda-el-p2p" || node_key) rather than reused. Both protocols use
secp256k1, but libp2p's Noise handshake and discv4 packet signatures plus RLPx
auth are unrelated, and one secret should not serve both. Deriving keeps the
identity deterministic across restarts, which the devnet script relies on.

The devnet script now proves propagation instead of assuming it: it submits to
exactly one node and asserts a *different* proposer included the transaction.
Inclusion alone would prove nothing, since the receiving node proposes eventually
regardless. To keep that deterministic it submits to the node that just proposed,
which is a full rotation away from proposing again. If the includer turns out to
be the submitter anyway, that is reported as gossip unproven rather than a
failure. --no-el-p2p restores isolated mempools and the fan-out submission.

Verified: 577 tests pass, fmt and clippy -D warnings clean. On a 3-node devnet all
three execution layers came up with distinct enodes and discv4 listening, a
transaction submitted to node 0 alone was included by node 1 at the first
opportunity, and that block reports gasUsed=21000. Chain finalized at slot 45 with
no synthetic fallbacks, rejected payloads or panics.

Blob (EIP-4844) transactions can now be submitted, included and executed. They
must arrive in the wrapped form, 0x03 || rlp([tx, wrapper_version, blobs,
commitments, proofs]) — the shape eth_sendRawTransaction takes — because the
mempool needs the sidecar to verify the KZG proofs and to build with later. The
bare form that appears inside blocks decodes to an empty bundle, so it is refused
with a message naming the wrapped encoding rather than an opaque bundle error;
sending the block form is the obvious mistake to make.

wrapper_version must be 0, one proof per blob. Version 1 is the EIP-7594 cell-proof
encoding for Osaka and later, and this EL genesis is Cancun. Current tooling often
defaults to version 1, so a transaction built by an up-to-date library is rejected
until told otherwise.

What this does NOT provide is data availability, and that is worth stating
plainly because the execution path works well enough to hide it.
ExecutionPayloadV3 has no sidecar field, so the blobs never cross the Lean
network; they exist only in the mempools that received the transaction, and the
eviction-on-import that transactions require in the first place discards those.
Nothing can retrieve them afterwards. BLOBHASH and the point-evaluation precompile
are unaffected, since they need only the versioned hashes and caller-supplied
data, so nothing inside the EVM notices. Treat blob support as exercising the fee
market and the execution path, not as a DA layer. Recorded as gotcha 6 in
docs/ethrex-inprocess-integration.md and on the endpoint in docs/rpc.md.

The load-bearing test is peer_executes_a_blob_block_without_ever_seeing_the_sidecar:
it builds a blob block on one engine and executes the payload on a second engine
that never saw the blobs. It passes because block validation derives blob gas and
count from blob_versioned_hashes alone and every KZG check lives on the
mempool-insertion path, never on import. That is what makes blob transactions safe
to include at all — had it failed, one blob transaction would fork the network — so
it is asserted rather than inferred from reading ethrex.

c-kzg is now declared explicitly on ethrex-blockchain. It already arrived through
ethrex-p2p's default features, but a c-kzg-gated API is called directly now, and
naming it keeps `cargo check -p ethlambda-ethrex-engine` and any
--no-default-features build honest.

ethrex-crypto joins the workspace for the KZG trusted-setup warm-up, pinned to the
same rev as every other ethrex crate and with default-features = false. That last
part is load-bearing: its defaults include kzg-rs, a second pure-Rust KZG
implementation nothing here uses, and because cargo unifies features across the
graph, enabling it once drags kzg-rs and its Plonky3 tree into every build — 468
lockfile lines, including a duplicate p3-util, which has broken this workspace's
MSRV before. Declared correctly the delta is a single dependency edge. ethrex
declares it the same way.

Verified: 580 tests pass, three of them new, fmt and clippy -D warnings clean. A
3-node devnet still finalizes with cross-node transaction gossip intact, confirming
the added feature and the warm-up do not disturb node startup.

…helper

Merging main brought in #552's `signed_block` test helper, written against a
`BlockBody` that has no `execution_payload`. This branch adds that field, so the
branch compiled and main compiled but their merge did not — a semantic conflict
with no textual conflict, which is why git reported the PR mergeable and only the
merge-commit build caught it.

The helper builds blocks for fork-choice tests that never reach the execution
layer, so the default all-zero payload is the right value.

Block import is gated on execution, so an execution layer that cannot extend the
consensus head is not degraded — it is terminal. Every arriving block names a
parent it does not have, fails, and is dropped before the store sees it. With the
execution store in memory, that was the state of every restarted node: consensus
resumed at its old slot, execution rewound to genesis, and the node never followed
the chain again. Checkpoint sync made it worse by moving the consensus head
further ahead.

Two parts. The execution store now persists in RocksDB under --data-dir/el, on the
same volume as the consensus store so it survives container recreation rather than
just a restart. And el_sync replays, at startup, whatever the execution layer is
missing: walk back from the consensus head to the last block it can actually
extend, then execute forward. The Lean chain carries every ExecutionPayloadV3 in
its block bodies, so it already is a complete execution history — no new wire
protocol and no peers needed.

Persistence turned out not to be sufficient on its own, which is worth recording:
a reopened datadir keeps the block index and canonical head but NOT executable
state. So the head is present and unbuildable, and the first attempt at gap
detection — "do I have this block?" — answered yes and concluded there was nothing
to do. The predicate is now can_build_on: the block is present AND its post-state
is reachable, which is the same condition ethrex checks before accepting a fork
choice update, and therefore the exact question build_payload will later ask.
Measured on a devnet restart: 44 payloads replayed in 8ms.

Also fixes a latent corruption risk found while testing this. The value seeded
into the consensus genesis anchor was read from head_hash(), which equals the
genesis hash only on a fresh store. With execution state persisted, a node whose
consensus datadir was wiped would have anchored a newly created genesis state to
whatever block its execution layer last executed, producing a chain no peer
agrees with — silently. Added genesis_hash() and used that.

start_p2p now runs after the resync rather than before. It calls set_synced(),
which is what unlocks inbound transaction gossip, and that should not happen while
the execution layer is still behind the chain.

The devnet harness gains --restart-node N. It stops a node, waits out the
gossipsub backoff, starts it, and then asserts what a node on a private fork
cannot fake: blocks imported from peers after the restart, and no synthetic-payload
fallbacks. The first version of this check asserted that the head slot advanced
and that the node was level with its peers — both of which a forked node satisfies
exactly, since a fork advances at the same rate. It passed while the node was
importing nothing. Head slot is now reported rather than asserted. Per-node checks
also count nodes instead of log lines, since a restart logs startup twice and
produced totals like "4/3 nodes".

Verified: 585 tests pass, three new, fmt and clippy -D warnings clean. On a 3-node
devnet, node 2 was stopped at slot 54 and restarted: it replayed 44 payloads,
imported 16 blocks from peers, built payloads with no synthetic fallbacks, and
reached slot 84 level with its peers. The chain finalized at slot 75 throughout,
with cross-node transaction gossip still working.

These ran the 16-node deployment on ethlambda-2 and lived in /tmp, which is the
wrong place for the only written record of how to launch this.

server-launch.sh exists because the launch is necessarily two-phase.
--el-bootnodes needs an enode URL that cannot be computed in advance: each node's
execution-layer key is a keccak derivation of its consensus node key, so the only
way to learn the enode is to ask the node. It starts node 0 alone, reads the enode
out of its log, and hands it to the rest; one bootnode is enough because discv4
finds the remainder of the mesh. It refuses to launch when the EL genesis has no
funded account, or when the subnet count disagrees with the genesis on disk —
either produces a devnet that looks healthy and cannot demonstrate anything.

server-demo-tx.sh submits one transaction to one node and reports which node
included it. It walks pre-signed transfers in nonce order and skips any already
spent, because the mempool rejects a replayed nonce and a demo that works exactly
once is not much of a demo. It submits to the node that just proposed — the one
furthest from proposing again — so inclusion by a *different* node is deterministic
rather than luck. That distinction is the whole point: inclusion alone would only
show that a mempool drains, while a different includer shows the transaction
travelled.

The generator for those transfers is an ignored test alongside the existing
fixture generator, parameterised by DEMO_TX_DIR and DEMO_TX_COUNT.

Verified in use: on the 16-node deployment, two consecutive runs submitted to
nodes 3 and 10 and were included by nodes 5 and 11 respectively, each with
gasUsed=21000, while the chain finalized normally.